Papers with knowledge evaluation
Cracking the Code: Multi-domain LLM Evaluation on Real-World Professional Exams in Indonesia (2025.naacl-industry)
Copied to clipboard
| Challenge: | Using the entire dataset, shuffling answer options introduces instability in the insurance and finance sectors. |
| Approach: | They propose a dataset for evaluation of performance in vocational and professional certification exams in Indonesia. |
| Outcome: | The proposed dataset includes 8,834 multiple-choice questions from 27 large language models across six key sectors. |
How Do Multilingual Language Models Remember Facts? (2025.findings-acl)
Copied to clipboard
| Challenge: | Prior research has focused on English monolingual models, but how these mechanisms generalize to non-English languages remains unexplored. |
| Approach: | They analyze three multilingual LLMs to find out how they can generalize recall mechanisms . they find that subject enrichment is language-independent, object extraction is language dependent . |
| Outcome: | The proposed model performs better in multilingual contexts than in English models . the model is more efficient in multi-lingual context, but it is more complex in multilinguistic models compared to English models. |
Take Care of Your Prompt Bias! Investigating and Mitigating Prompt Bias in Factual Knowledge Extraction (2024.lrec-main)
Copied to clipboard
| Challenge: | Recent research shows that pre-trained language models suffer from “prompt bias” in factual knowledge extraction. |
| Approach: | They propose a representation-based approach to mitigate prompt bias during inference time by querying the model and removing it from its internal representations to generate debiased representations. |
| Outcome: | The proposed approach corrects the overfitted performance caused by prompt bias and significantly improves prompt retrieval capability. |